Simulated Students and Classroom Use of Model-Based Intelligent 

Tutoring 


by 

Kenneth R. Koedinger 


45 





Simulated Students & Classroom 
Use of Model-Based Intelligent 

Tutoring Systems 


Kenneth R. Koedinger 

Human-Computer Interaction 

& Psychology 

Carnegie Mellon University 


CMU Director of the 
Pittsburgh Science of Learning Center 


Email: K o cti i n r(« emu.edu 


Phone: 412-268-766 





Modeling & simulation to 
enhance education 

• Two paths: 

1. Students create models & use simulations 

2. Researchers create models of learners to 
guide materials development 

• #1 is great way to potentially enhance 
learning, however, 

• Understanding student learning (#2) is 
critical to effective design & use 




Real World Impact of HCI & 
Learning Technologies 

Algebra Cognitive Tutor 

• Based on computational models of 
student thinking & learning 

• Course used nation wide 

-Over 4000 schools, 35 states, 475K 
students use for 80 minutes per week 

• Spin-off: 



Koedinger, Anderson, Hadley, & Mark (1997). 
Intelligent tutoring goes to school in the big city. 




Overview 


• Background: Cognitive Tutors 

- Simulating tutoring 

- Data crucial to create accurate model 

• Pittsburgh Science of Learning Center 

- Generalizing learning science & technology 

• Examples of advanced modeling efforts 

- Simulating student learning 

- Modeling metacognition, help-seeking 

- Machine learning detectors of student engagement 




scenario 



Changing axis bounds 
Changing axis intervals 
Correctly pteeing points 
Wf is expression, any form 
Find Y, any form 
Find X, any form 
identifying units 
Entering a givers 


[you have entered the given 0 in the 





Multi-Disciplinary Approach 




Cognitive Tutor Technology: 

Use ACT-R theory to individualize instruction 

• Cognitive Model: A system that can solve problems 
the various ways students can 

Strategy 1: IF the goal is to solve a(bx+c) = d 

THEN rewrite this as abx + ac = d 
Strategy 2: IF the goal is to solve a(bx+c) = d 

THEN rewrite this as bx + c = d/a 
Misconception: IF the goal is to solve a(bx+c) = d 

THEN rewrite this as abx + c = d 



Cognitive Tutor Technology: 

Use ACT-R theory to individualize instruction 


• Cognitive Model: A system that can solve problems in 
the various ways students can 



If goal is solve a(bx+c) = d 
Then rewrite as abx + c = d 


If goal is solve a(bx+c) = d 
Then rewrite as bx+c = d/a 


6x - 15 = 9 2x - 5 = 3 6x - 5 = 9 


• Model Tracing: Follows student through their individual 
approach to a problem -> context-sensitive instruction 



Cognitive Tutor Technology: 

Use ACT-R theory to individualize instruction 


• Cognitive Model: A system that can solve problems in 
the various ways students can 


If goal is solve a(bx+c) = d 
Then rewrite as abx + ac = d 

Hint message: “Distribute a 
across the parentheses.” 

Known? = 85% chance 

6x - 15 = 9 


3(2x - 5) = 9 

^ X If goal is solve a(bx+c) = d 

> Then rewrite as abx + c = d 

Bug message: “You need to 
multiply c by a also.” 

Known? = 45% 

2x - 5 = 3 6x - 5 = 9 



• Model Tracing: Follows student through their individual 
approach to a problem -> context-sensitive instruction 


m 


Knowledge Tracing: Assesses student's knowledge 
growth -> individualized activity selection and pacing 



Research base 

'Cognitive 

Psychology 

Artificial 

Intelligence 


uumculum Content 

Math Instructors 
Math Educators 
NCTM Standards 


Geometry Algebra II 


Algebra I 


Equation 

Solver 






What prior knowledge do algebra 
students have? 

Which problem type is most difficult for beginning Algebra 

students? 

Story Problem 

As a waiter, Ted gets $6 per hour. One night he made $66 in 
tips and earned a total of $81 .90. How many hours did Ted 
work? 


Word Problem 

Starting with some number, if I multiply it by 6 and then add 
66, I get 81 .90. What number did I start with? 


Equation 


x*6 + 66 = 81.90 




Percent Correct 


Algebra Student Results: 
Story Problems are Easier! 


80% i 


60% i 


40% - 
20 % - 

0 % - 


70% 



Story Word Equation 

Problem Representation 


Koedinger, K. R. & Nathan, M. J. (2004). The real story behind story problems: Effects of 
representations on quantitative reasoning. The Journal of the Learning Sciences , 13 (2), 129-164. 



Practical & Theoretical Implications 
of Surprising Results 

• Guided Cognitive Tutor Algebra design 

- Success due in part to smoothly bridging 
from students' existing common sense 

• Inspired basic cognitive modeling work 
to explain these results 

- Coded student solutions for alternative 
strategies & errors 

- Model could generate both & fit student 
data on frequency of both 
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Integrated tutor, text, and teacher training 
In computer lab 2 days/week, classroom 3 days/week 

Learn by doing: 

- Project-based 

- Student-centered 

- Cooperative learning 

- Teacher as facilitator 



Original Field Study Results 


• Full year classroom experiments with comparison classes 

• Replicated over 3 years in urban schools 


• In Pittsburgh 
& Milwaukee 

• Results: 

50-100% better on 
problem solving & 
representation use. 

15-25% better on 
standardized tests. 


Koedinger, Anderson, Hadley, & 
Mark (1997). Intelligent tutoring 
goes to school in the big city. 
International Journal of Artificial 


^Traditional Algebra Course 
□ Cognitive Tutor Algebra 





Iowa 


SAT subset Problem Represent- 
Solving ations 


Intelligence in Education. 





Many other studies of 
Cognitive Tutor Algebra 

• 11 study reports available 

- From 1994 to present, 11 different districts 

- More than 8000 students in these studies 

- Most run independently of Carnegie 

• Significant positive results in all but 1 case 
(which was a tie) 

• See www.carnegielearning.com/results/reports 




Overview 


• Background; Cognitive Tutors 

- Simulating tutoring 

- Data crucial to create accurate model 

• Pittsburgh Science of Learning Center 

- Generalizing learning science & technology 

• Examples of advanced modeling efforts 

- Simulating student learning 

- Modeling metacognition, help-seeking 

- Machine learning detectors of student engagement 




Transition to Pittsburgh 
Science of Learning Center 

Past Success: 

• Cognitive Tutors as delivery vehicle 

- Bring existing Learning Science to 
classroom 

New Goal: 

• Cognitive Tutors as research platform 

- Create new Learning Science & Technology 

• 5 year, $25 million research center: 




Research base 


Cognitive & 
Educational 
Psychology 

Artificial 

Intelligence 


wnfirMimYiWtfsiiiEnn 


University 
Instructors & 
Professors 


Statistics 


Philosophy Economics Chemistry 








Pittsburgh Science of 
Learning Center (PSLC) 

• Problem : Inadequate theory to engineer courses 
to be provably effective in raising student 
achievement 

• Solution: PSLC's theory & facility development 

- Theory: Unified effort toward robust learning theory 

- Facility: Learn Lab 's courses, technology, DataShop 

• Scientific merit & broad impact 

- Advance a practical learning theory, evidence-based 
education, fast & natural dissemination 



PLSC Focus on Robust 
Learning 


Other Intelligent Tutoring Systems yield better 
learning on immediate post-tests 

- Woolf, Graesser, VanLehn, ours ... 

Push to address robust learning 

- Transfer beyond isomorphic probs 

- Long term retention 

- Preparation for better future learning 

Address both sides of ed wars 

- Basic fluency & deep conceptual understanding 

Tutor at meta-cognitive level 



PSLC s Resources 


• 7 Technology-enhanced courses where 
researchers can run studies 

- Algebra, Geometry, Chemistry, Physics, 
French, Chinese, English 

• Data Shop 

- A repository of student learning data sets 

- Reporting, export, & analysis tools 

• Tools for authoring tutors ... 



o\ 
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Learn Lab: 7 testbed 
courses open for studies 


French Gulture Tutor 


utor lab , 


Bear why you chsn< chs :X rspo- 


Rernsnt>ar, you can rewind and review the vide 
Please hit ENTER to record your responses. 


Whet do you fchr.k he wl respond? 


What do you think might be 3 Ikeiy respoo 
in your culture? 


A 2000 -kg car in neutral at 

driveway 20.0 m long slips its p^m^rak^nd rolls down. 
Assume that the driveway is frictionless. 

What is the magnitude of the velocity of the car when it 
hits the garage door? 


Answer: 

+Y 






Technology-enhanced 

courses: 

- 2 math, 2 science, 

3 language courses 

Tutors, simulations, 
video, chat rooms, 
multimedia ... 



Overview 


• Background: Cognitive Tutors 

- Simulating tutoring 

- Data crucial to create accurate model 

• Pittsburgh Science of Learning Center 

- Generalizing learning science & technology 

• Examples of advanced modeling efforts 

- Simulating student learning 

- Modeling metacognition, help-seeking 

- Machine learning detectors of student engagement 




: Tutors have been sueces 
students' math test scor 
to! and middle-school 
is, but their development 
illy required eonsiderabls 
expertise. With the Cop 

h'%* ,-»****-,. ^ 


Cognitive Tutor Authoring 
Tools (CTAT) 

• Easier authoring of Tutoring Systems 

- Non-programmer methods 

- General plug-and-play architecture 

• Tutors created in a variety of domains 

- Chemistry, Thermodynamics, Genetics, 
Law, French culture ... 




Aids for Building Cognitive 
Tutors 

• Iterative design-&-test for GUI 

- Building, testing, and modifying a prototype 

- Cycling quickly and easily 

• Cognitive Modeling 

- Generating a cognitive model without 
programming 

- Human friendly testing & debugging 



Solution 


• Integrated intelligent authoring 
environment 

- CTAT : Cognitive Tutor Authoring Tools 

- Simulated Student: Machine learning agent 
that learns cognitive skills 




CTAT: Cognitive Tutor 
Authoring "ools 


Built ttiioi 
Chemisin 


Jfcdwvsor Recorder for 




Overview 


• Background: Cognitive Tutors 

- Simulating tutoring 

- Data crucial to create accurate model 

• Pittsburgh Science of Learning Center 

- Generalizing learning science & technology 

• Examples of advanced modeling efforts 

- Simulating student learning 

- Modeling metacognition, help-seeking 

- Machine learning detectors of student engagement 



SimStudent 

• Work with Noboru Matsuda & William 
Cohen 

• Learn production rules by 
demonstration 

• 3 parts: What, how, and when 

- What to operate on 

- How to operate 

- When to do it 


Structure of a Production Rule 


if 


such and such constraints hold 

When 


among this and that GUI elements 

What 

Then 


Left Hand Side 
(IMS) 


Right Hand S'nk 


do actions with the GUI elements 

How 



Structure of a Production Rule 


(defrule trans-lr-lhs 


rums Memory 


viiz 


/ ?problem <- ( (problem ( interface-elements ?tablel ? ? ?))} 
t ?tablel <- (table (columns ?columnl ) ) 

?columnl <- (column (cells $?ml ?cellO $?)) 

?cellO <- (cell (value ?valO&~nil ) ) 

■ Pproblem <- (problem (interface-elements ? ?table2 ? ?)) 

Ptable2 <- (table (columns ?column2 ) ) 

I ?column2 <- (column (cells $?m2 Pcelll $?)) 

?celll <- (cell (value ?vall&-nil) ) 


GUI elements 

WME path 


if 


4f 


IMS 8 




?columnl <- (column (cells $Pm3 ?cell2 $?)) 

?cell2 <- (cell (name Pselection) (value Pinput)) 


(test ( consecutive-row PcellO ?cell2 ) ) 
(test (same-column PcellO ?cell2 ) ) 
(test (distinctive PcellO ?cell2 ) ) 
(test (consecutive-row Pcelll ?cell2 ) ) 
(test (same-column Pcelll ?cell2 ) ) 
(test (distinctive Pcelll ?cell2 ) ) 


Topological 

amstramti 


j 

Wsm m m lipiii m 


(test (polynomial PvalO ) ) 

(test (not (has-var-term ?vall ) ) ) 


. _ 

I Feature 




constraints 


| WME 

| I'fJfUihs 


i 


t 




Constraints 


(bind ? input ( f irst-var-term PvalO) ) 
(modify Pcell2 (value ? input ) ) ) 


mi 





Learning Techniques 

(defrule trans-lr-lhs 

. ^ Memory ^wMEi v 

| Pproblem <- ( {problem (interface-elements Ptablel ? ? ?))) ““j WME path 
Ptablel <- (table (columns ?columnl)) 

| ? column 1 <- (column (cells $?ml ?cellO $?)) 

PcellO <- (cell (value ?valO&~nil ) ) 

?problem <- (problem (interface-elements ? ?table2 ? ?)) 

?table2 <- (table (columns ?column2)) 

?column2 <- (column (cells $?m2 ?celll $?)) 

?celll <- (cell (value ?vall&~nil)) 

?columnl <- (column (cells $?m3 ?cell2 $?)) 

?cell2 <- (cell (name ?selection) (value ?input) ) 



=> 


(test (consecutive-row PcellO ?cell2 ) ) 
(test (same-column ?cellO ?cell2)) 
(test (distinctive ?cellO ?cell2)) 
(test (consecutive-row Pcelll ?cell2)) 
(test (same-column ?celll ?cell2)) 
(test (distinctive ?celll ?cell2)) 

(test (polynomial ?valO)) 

(test (not (has-var-term ?vall))) 


Topological 

constraints 





constraints 1 


WME 

ctmdkkms 


(bind Pinput ( f irst-var-term PvalO)) 
| (modify ?cell2 (value ? input) ) ) 


1 Brute Force Search 


of attention 



~j RHS 





SimStudent Goals & Progress 


Improving tutor authoring 

- Works in multiple domains 

• Past: Multi-column addition & multiplication, Tic-Tac- 
Toe, fraction addition 

• Current: Equation solving, Stoichiometry 

• Future: Scientific reasoning 

Simulating human learning 

- Studies varying: 

• Alternative curriculum sequences 

• Human-like memory limitations 

- Surprising result: Hard-to-easy curriculum sequence 
better than easy-to-hard 




Tutoring Help-Seeking 


Roll, Aleven, McLaren, Ryu, Baker, & Koedinger, (2006). The help tutor: 
Does metacognitive feedback improve students' help-seeking actions, skills 
and learning? In Proceedings of the 8th International Conference on 
Intelligent Tutoring Systems, 



oo 
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seeking behavior 


Production system 
implementation 

About 50 rules 


Use same model 
tracing technique 

But at the 

"metacognitive" level 

- In addition to 
geometry tutor at 
cognitive level 


Ask for hint 


Try step (i,e , [ 
cntef answc*) I 
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Tutoring Help-Seeking 


Goal: Foster long-term learner 
independence / 

Model of desired learning 
& help-seeking behaviors 


START 


• Provide tutoring 
relative to this mode 






Overview 

• Background: Cognitive Tutors 

- Simulating tutoring 

- Data crucial to create accurate model 

• Pittsburgh Science of Learning Center 

- Generalizing learning science & technology 

• Examples of advanced modeling efforts 

- Simulating student learning 

- Modeling metacognition, help-seeking 

- Machine learning detectors of student engagement 



Another c xample of using ML 

Adapting to When Students 
Game an Intelligent Tutoring 
System 


Former Phd student Ryan Baker 



Gaming the System 


Attempting to get correct answers and advance 
in a curriculum by taking advantage of the 
software's help or feedback, rather than by 
actively thinking through the material" 


For instance 

- Systematic Guessing 

(cf. Mostow et al 2002) 

- Drilling through Hints 

(cf. Wood & Wood 2000 
Aleven 2001) 



Scooter the Tutor 


A tutor agent 

Uses a machine learning "detector" 
to recognize student gaming 
behavior 

Intended to reduce gaming & 
negative consequences for learning 



oo 
V. O 



Gaming detector 
construction 


1. Collect data 

- Observe students in 
computer lab 

• Code off-task behaviors 

- Get tutor interaction log 
data from same sessions 


mm 
m m 


2. Train a machine learning system using data 

- Techniques: Fast correlation-based filtering, forward 
selection on log data variables & interaction terms 

- Generalized & cross-validated across four tutor 
domains 


scenario 


v. O 
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Samantha Is trying to find out what brand of dog food her dog 
Champ likes best Each day she feeds him a different brand and 
sees how many bowls he eats. But then her mom says that maybe 
her dog just eats more on days when he exercises more. 

Please draw a scatterplotto show how many bowls the dog eats, 
given the dog's level of exercise that day. 



J Understanding categorical variables 
J Understanding numerical variables 
J Determining whether a variable can be used in a k 
J Placing the Independent variable on the X axis 
J Placing the dependent variable on the Y axis 
J Choosing the lower bound of an axis 
J Finding the smallest value in a data set 
J Finding the largest value in a data set 
Fining The range of a data set 
Choosing an appropriate scale 
Labeling the first value on the axis 
J Labeling second value on the axis 
Labeling subsequent values on the axis 
Plotting the first point 
Plotting subsequent points 




Exercise 

(minutes) 

Bowls 

Btbbles 
a n d Nits 

12 


Barleys 

25 

0 

PappV 

Oftew 

28 

3 

Unleashed 

5 

0 

Premium 

18 

1 

Busied 

14 

1 

Delight 

10 

1 

T ucke/s 
Regular 


3 

MM Deg 

IS 

2 

No Bones 
About it 

21 

2 































During the Student's Tutor 
Use 

• Scooter responds to gaming in two 
ways 

- Emotional expressions 
— Supplementary exercises 


V. O 

K> 



Emotional Expressions 

• If the student never games, Scooter 
looks happy 


V. O 



Emotional Expressions 

• If the student appears to be gaming, 
Scooter looks increasingly displeased 
and becomes redder and redder 


|||i 



Supplementary exercises 


,» Tntnr 


Multiple Levels 

If a student is 
wrong, receives 
another question 


Why do we need to kno w the 
largest value of height 
in this part of the problem? 


j To help pick a scale for the axis 
f To help pick the last label of the axis 

! To help pick the first label of the axis 

f” To help pick which point to plot first 



Scooter results 


• Reduces gaming behavior 

• Supplementary exercises increase 
learning 

• Emotional responses do not 


Baker et al. (2006). Adapting to When Students Game an Intelligent 
T utoring System (Best Paper at ITS06) 




Two kinds of student 
modeling approaches 


1. Rational approach 

- Analyze domain & code model that "makes sense" 

- Example: Help-seeking model 

2. Empirical approach 

- Collect human data driven & use statistical machine 
learning to learn model 

- Example: Gaming detector 


Remember PSLC's resources 

- One is DataShop 

- It provides data needed for empirical approach 



V. O 
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DataShop: Get data to build 
or test models of learning! 


See LearnLab.org 


Microgenetic 
log data of 
student 
learning over 
semester 

Data from 

math, 

science, 

language 

courses 


Sam ples g 

My Samples 

Stes&Sampfes 
g All Data 


Learning Curvt 

mwm 

■* Kncwtedga Component 
Student 
Type 

g Assistance Scars 
>; Error Rats 

v View Predicted 


K nerMedg e Compo nent Model s j 
Primaiy '' Tcxtboo^J^ 
Secondary ' ~'HP 


Knowledge Components 
deselect ail ! select al 

g circle-area. 

□ ctrcte-circurrie. 

□ circte-diamsier 

0 oompose-by-addL.. 
g eompos$^-mulL«. 
0 equRjI-heighi? 

□ parallelogram-area 

□ peniagcmarea 


circle-area 


opportunity 

AH Data - -All Data - Textbook (Predicted) j 


j Ppporairfty Nestor 1 234 SST 3 S 10 

1 NafPber c>( Cbservatfors 28 2B 2B 22 23 Zz 23 2? 28 23 


All Selected K.C.S 


circle -area 


2 4 6 8 :S 


coiJipD5e~bv-multiplication 


t om p os e -fey-add It i on 




rectaofl€~arsi 



Summary 

• Two educational uses of models & simulations 

1. Students create models & use simulations 

2. Researchers create models of learners to guide 
development of reliably effective materials 

• Cognitive Tutors simulate & support tutoring 

- Data is crucial to create effective model 

• Pittsburgh Science of Learning Center 

- Resources for modeling, authoring, experimentation 

- Repository of data & theory 

• Examples of advanced modeling efforts 

- SimStudent learns rule-based model 

- Help-seeking model: Tutors metacognition 

- Scooter uses machine learning detectors of student 
engagement 



Pittsburgh Science of Learning 
Center opportunities 


Propose a classroom study 
or attend summer school 

Analyze student data 

- TagHelper: Verbal 
data coding software 

- DataShop: Data sets, 
reporting & analysis tools 

Author a tutor or 
on-line activity 

- Cognitive Tutor Authoring Tutors 

- TuTalk: Authoring tutorial dialog 

- Open Learning Initiative 

- On-line assessments 


LearnLab.org 


ctat.pact.cs.cmu.edu 

www.cmu.edu/oli I 

ass1stment.org I 







